Method, a device and a system for checkout

ABSTRACT

The present invention discloses a method, a device and a system for checkout, and belongs to the technical field of computer. The checkout method comprises: identifying a pre-registered customer to acquire an identity information of the customer, the identity information containing face data and a payment account of the customer; tracking the customer whose identity information has been acquired, in a shopping place in real time, and acquiring a position of the customer; judging whether the position of the customer is consistent with a position of an item in the shopping place, and if so, associating the customer with a take-up action or a put-back action aiming at the item, and generating a shopping list of the customer after identifying the take-up action or the put-back action, and the item at which the take-up action or the put-back action aims; and performing checkout of the shopping list. The present invention further discloses a checkout device comprising a top camera, a forward camera, a downward camera and a server. The present invention further discloses a checkout system comprising a client and a checkout device. According to the above-mentioned technical solutions of the present invention, the situations that workload is heavy due to the adhesion of Radio Frequency identification (RFID) tags and RFID tags are easily be damaged may be avoided.

BACKGROUND Field of Invention

The present invention belongs to the technical field of computer, andparticularly relates to a method, a device and a system for checkout.

Background of the Invention

When a customer saw an item he likes or requires in a shopping place,such as a supermarket, a store and the like, he needs to performcheckout with an operator of the shopping place to obtain the item.Generally, the checkout is completed in the customer queuing mannerbeside a cashier counter.

With the development of the artificial intelligence technology, thecheckout mode is also changing. For example, after walking into theshopping place and selecting an item, the customer can perform checkoutand leave immediately without queuing and waiting at the cashiercounter, which is commonly known as the checkout mode of “Just WalkOut”.

In the prior art, the solution of “Just Walk Out” is implemented mainlyon the basis of the Radio Frequency identification (RFID) technology.Upon application, a small radio frequency module which does not requirea battery is adhered to the item. When the item is passed through acheckout counter (or a checkout area) provided with a RFID detectiondevice, the checkout counter can emit a wireless signal to the item.After receiving the signal, the small radio frequency module feeds backa signal carrying ID information of the item to the checkout counter,and the checkout counter generates a bill for checkout according to theID information. The method has the following defects. because the smallradio frequency module is necessary to be adhered to each item, theworkload is extremely heavy for workers in the shopping place and thecost is high. Moreover, if the small radio frequency module falls fromthe item or is damaged naturally or artificially, the checkout countercannot identify the item, resulting in a loss for a merchant. Inaddition, if a RFID is adhered to some metal items, a signal shieldingproblem may be occurred.

SUMMARY

In order to solve the problems that the workload for adhering RFID tagsis heavy and RFID tags may be easily damaged in the prior art, in oneaspect, the present invention provides a checkout method, comprising thesteps of: (S1) identifying a pre-registered customer to acquire anidentity information of the customer, the identity informationcontaining face data and a payment account of the customer; (S2)tracking the customer whose identity information has been acquired, in ashopping place in real time, and acquiring a position of the customer;(S3) judging whether the position of the customer is consistent with aposition of an item in the shopping place, and if so, associating thecustomer with a take-up action or a put-back action aiming at the item,and generating a shopping list of the customer after identifying thetake-up action or the put-back action, and the item at which the take-upaction or the put-back action aims; and (S4) performing checkout of theshopping list.

In the checkout method as mentioned above, preferably, in the step (S3),judging whether the position of the customer is consistent with theposition of the item in the shopping place specifically comprises:representing the position of the item with a position of a forwardcamera which is mounted on a shelf for bearing the item and is used forshooting forwards, in case that the identity information of the customerrepresented by an image containing the customer, which is shot by theforward camera, is the same as the identity information acquired in thestep (S1), judging that the position of the customer is consistent withthe position of the item in the shopping place.

In the checkout method as mentioned above, preferably, in the step (S3),identifying the take-up action or the put-back action specificallycomprises: acquiring a plurality of frames of consecutive hand images ofthe customer in front of the shelf for bearing the item, andestablishing a motion track of a hand for the plurality of frames ofconsecutive hand images on a timeline, in case that it is detected thatthe motion track of the hand is an inward movement from the outside of apredetermined virtual action boundary and the item is taken in the hand,identifying the action as the put-back action; in case that it isdetected that the motion track of the hand is an outward movement fromthe inside of the virtual action boundary and the item is taken in thehand, identifying the action as the take-up action, wherein the outsideof the virtual action boundary is in a direction away from the shelf,and the inside of the virtual action boundary is in a direction close tothe shelf.

In the checkout method as mentioned above, preferably, in the step (S3),identifying the item at which the take-up action or the put-back actionaims specifically comprises the steps of: (S31) performing targetdetection on the plurality of acquired frames of hand images containingthe item to obtain a plurality of rectangular area imagescorrespondingly, wherein the rectangular area images are imagescorresponding to rectangular areas containing the item, and theplurality of frames of hand images corresponds to a plurality of camerasin a one-to-one manner; (S32) acquiring a plurality of primaryclassification results correspondingly, according to the plurality ofrectangular area images and a pre-trained first-level classificationmodel, and acquiring a first-level classification result of theplurality of frames of hand images, according to the plurality ofprimary classification results and a pre-trained first-level linearregression model, wherein the pre-trained first-level classificationmodel is a model that is constructed by an image identificationtechnique of convolutional neural network and trained by all the itemsin the shopping place; (S33) using the first-level classification resultas a first classification result; and (S34) using the firstclassification result as a to-be-identified item.

In the checkout method as mentioned above, preferably, after the step(S32) and before the step (S34), the method further comprises the stepof: (S35) obtaining a plurality of secondary classification resultscorrespondingly, according to the plurality of rectangular area imagesand a pre-trained second-level classification model, acquiring asecond-level classification result of the plurality of frames of handimages, according to the plurality of secondary classification resultsand a pre-trained second-level linear regression model, and using thesecond-level classification result as the first classification result,in case that the first-level classification result is a similar item,wherein the second-level classification model being a model that isconstructed by the image identification technique of convolutionalneural network and trained by items in a similar item group in theshopping place in advance; otherwise, executing the step (S33).

In another aspect, the present invention further provides a checkoutdevice, comprising: a registration module configured to receive identityinformation inputted by a customer upon registration and acquire theidentity information of the customer who intends to enter a shoppingplace; a real-time tracking module configured to be connected with theregistration module and configured to track the customer, whose identityinformation has been acquired by the registration module, in theshopping place in real time, and acquire a position of the customer; ashopping list generation module configured to be connected with thereal-time tracking module and configured to judge whether the positionof the customer acquired by the real-time tracking module is consistentwith a position of an item in the shopping place, and if so, associatethe customer with a take-up action or a put-back action aiming at theitem, and generate a shopping list of the customer after identifying thetake-up action or the put-back action and the item at which the take-upaction or the put-back action aims; and a checkout module configured tobe connected with the shopping list generation module and configured toperform checkout of the shopping list generated by the shopping listgeneration module.

In the checkout device as mentioned above, preferably, the shopping listgeneration module comprises: an association unit configured to representthe position of the item with a position of a forward camera, which ismounted on a shelf for bearing the item and used for shooting forwards,in case that the identity information of the customer represented by animage containing the customer, which is shot by the forward camera, isthe same as the identity information acquired by the registrationmodule, judge that the position of the customer is consistent with theposition of the item in the shopping place; an action identificationunit configured to acquire a plurality of frames of consecutive handimages of the customer in front of the shelf for bearing the item, andestablish a motion track of the hand for the plurality of frames ofconsecutive hand images on a timeline, in case that it is detected thatthe motion track of the hand is an inward movement from the outside of apredetermined virtual action boundary and the item is taken in the hand,identify the action as the put-back action; in case that it is detectedthat the motion track of the hand is an outward movement from the insideof the virtual action boundary and the item is taken in the hand,identify the action as the take-up action, wherein the outside of thevirtual action boundary is in a direction away from the shelf, and theinside of the virtual action boundary is in a direction close to theshelf; an item identification unit configured to identify the item atwhich the take-up action or the put-back action aims; and a shoppinglist generation unit configured to generate the shopping list of thecustomer, according to the identity information of the customerdetermined by the association unit, the take-up action or the put-backaction identified by the action identification unit, and the item atwhich the take-up action or the put-back action aims and identified bythe item identification unit.

In the checkout device as mentioned above, preferably, the itemidentification unit comprises: a target detection subunit configured toperform target detection on the plurality of frames of hand imagescontaining the item which are acquired by the action identificationunit, in order to obtain a plurality of rectangular area imagescorrespondingly, wherein the rectangular area images are imagescorresponding to rectangular areas containing the item, and theplurality of frames of hand images correspond to a plurality of camerasin a one-to-one manner; a first classification subunit configured toacquire a plurality of primary classification results correspondingly,according to the plurality of rectangular area images and a pre-trainedfirst-level classification model, and acquire a first-levelclassification result of the plurality of frames of hand imagesaccording to the plurality of primary classification results and apre-trained first-level linear regression model, wherein the pre-trainedfirst-level classification model is a model that is constructed by animage identification technique of convolutional neural network andtrained by all the items in the shopping place; a confirmation unitconfigured to use the first-level classification result as a firstclassification result; and a result determination unit configured to usethe first classification result as a to-be-identified item.

In yet another aspect, the present invention further provides a checkoutdevice, comprising: a top camera configured to shoot downwards from thetop of a shopping place to track a customer, whose identity informationhas been acquired, in the shopping place in real time; a forward cameraconfigured to shoot towards the front of a shelf to acquire an image ofthe customer positioned in front of the shelf for bearing an item; alower camera configured to shoot downwards to acquire a hand image ofthe customer; a processor; and a memory that recordsprocessor-executable instructions, wherein the processor is configuredto identify the pre-registered customer to acquire the identityinformation containing face data and a payment account of the customer,control the top camera to track the customer, whose identity informationhas been acquired, in real time and acquire a position of the customer,judge whether the position of the customer is consistent with a positionof the item in the shopping place, which is acquired by controlling theforward camera, and if so, associate the customer with a take-up actionor a put-back action aiming at the item, generate a shopping list of thecustomer after identifying the take-up action or the put-back action andthe item at which the take-up action or the put-back action aimsaccording to the hand image acquired by the lower camera, and performcheckout of the shopping list.

In still a further aspect, the present invention further provides acheckout system, comprising: a client terminal configured to receiveidentity information inputted by a customer upon registration and sendthe identity information to a checkout device, and configured to receivea shopping list issued by the checkout device; and the above-mentionedcheckout device.

The embodiments of the present invention bring out the followingbeneficial effects by the above-mentioned technical solutions.

The operation cost is low, since the workload caused by adhesion of RFIDis saved, compared with a RFID solution. The application range is wide,since the present invention is applicable to any item, and is notrestrained by attributes of the item, such as forms, materials and thelike. The user experience is good, since the customer can acquirecorresponding information immediately after taking an item.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of the checkout method provided in anembodiment of the present invention;

FIG. 2 is a schematic flowchart of an image identification method basedon the convolutional neural network provided in an embodiment of thepresent invention;

FIG. 3 is a schematic flowchart of another image identification methodbased on the convolutional neural network provided in an embodiment ofthe present invention;

FIG. 4 is a structural schematic diagram of the checkout device providedin an embodiment of the present invention; and

FIG. 5 is a structural schematic diagram of a shelf for a checkoutdevice provided in an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In order to make objects, technical solutions and advantages of thepresent invention more apparent, the embodiments of the presentinvention will be further described in detail below in connection withthe drawings.

An embodiment of the present invention provides the checkout method.With reference to FIG. 1, the method comprises the following steps of S1to S4.

In the step S1, a pre-registered customer is identified to acquireidentity information of the customer.

Specifically, before the customer enters a shopping place, such as astore, a supermarket, an Application (App) corresponding to the checkoutmethod needs to be installed on mobile communication equipment of thecustomer, such as a mobile phone and a tablet personal computer, so asto complete registration of the customer. The identity (ID) informationof the customer, which is acquired during registration, includes but isnot limited to face data and a payment account of the customer. Thepayment account may be a bank account and may also be a third-partypayment account, such as Alipay, WeChat Pay, QQ Wallet, JD Wallet, andthe like, and the identity information may also include the name, mobilephone number, ID number, gender and profession. The registration of thecustomer may also be completed by an applet (or called as WeChat applet)corresponding to the checkout method in WeChat. The registration of thecustomer may also be completed by subscribing a WeChat official accountcorresponding to the checkout method. Then, the customer scans atwo-dimensional code at a doorway of the shopping place by the App orthe applet or the official account, or a store scans a two-dimensionalcode generated by the App on the mobile communication equipment held bythe customer to implement verification on the ID of the customer, i.e.,identify that the customer is one of the registered users, so as toacquire the identity information corresponding to the customer, in otherwords, it is known who is the customer entering the shopping place.After the identity information of the customer is acquired, a door lockof the shopping place may be unlocked automatically, then the customermay open a door by pushing the door inwards or pulling the door outwardsor pushing the door horizontally, and enter the shopping place to selectitems. If the customer has not completed registration, the customer canbe identified as a non-registered user, and the door lock in theshopping place remains locked, so the customer cannot enter the shoppingplace. In other embodiments, a biometric identifier, such as afingerprint identifier and a face identifier, can be arranged at thedoorway of the shopping place to identify the ID of the customer in theshopping place by biometric identification technology. When the customerregisters on the App, the biological data of the customer, such asfingerprint data, may be collected. The door of the shopping place maybe opened by a worker after the identity information of the customer isacquired.

In the step S2, the customer whose identity information has beenacquired is tracked in the shopping place in real time, so as to acquirea position of the customer.

When the customer enters the shopping place, his identity is necessaryto be identified. The identified customer freely moves and takes up orputs back items in the shopping place. If it is required to master theidentity information of each customer in the shopping place at anymoment, the customer in the shopping place needs to be continuouslytracked after the identity of the customer entering the shopping placeis confirmed, i.e., the customer needs to be positioned in real-time.

Specifically, a top camera is deployed at the top in the shopping place.The top camera can shoot a video stream of movement of the customer inthe shopping place. By analyzing the video stream and comparing thedifference between the adjacent frames in the video stream, real-timepositioning of the customer in the shopping place is implemented. Aplurality of top cameras can be deployed on a ceiling so as to enable ashooting range to cover the overall store, and a shooting direction ofthe top camera is slantingly downward. When the customer enters thestore, the identity of the customer is identified. After the customerentered the store, the top camera on the ceiling carries out shootingfrom top to bottom to acquire an image of the customer in real time andbinds the image with the identity information of the customer, i.e., theidentity information of the customer moving in the store is known. Withthe movement of the customer in the different positions of the shoppingplace, other top cameras can always keep tracking the customer so as toposition the customer in the shopping place.

In the step S3, if the position of the customer is consistent with aposition of an item in the shopping place, the customer is associatedwith a take-up action or a put-back action aiming at the item; and ashopping list of the customer is generated, after identifying the itemat which the take-up action or the put-back action aim.

Specifically, after entering the store, the customer may move in thestore. When seeing the item the customer likes, the customer may stay infront of a shelf for bearing the item, and then carry out a take-upaction on the item, which shows that the item belongs to ato-be-purchased item, or carry out a put-back action, which shows thatthe item does not belong to a to-be-purchased item. The current positionof the customer can be acquired by step S2. If the current position isconsistent with the position of the item, a person performing thetake-up action or the put-back action aiming at the item on the shelf ismarked as the customer, in other words, the customer is associated withthe take-up action or the put-back action aiming at the item on theshelf, so as to know which customer carries out the take-up action orthe put-back action aiming at the item on the shelf in front of theshelf. After the item at which the take-up action or the put-back actionaims is identified, a shopping list corresponding to the customer can begenerated. The position of the item can be represented with a positionof a camera which is arranged on the shelf and used for shootingforwards. When the customer moves to the front of the shelf to select anitem, the camera for shooting forwards can shoot an image of thecustomer. If customer information contained in the image is consistentwith the identified identity information of the customer, it isdetermined that the position of the customer is consistent with theposition of the item. When the customer applies the take-up action onthe item, the item can be correspondingly added in the shopping list ofthe customer; and when the customer applies the put-back action to theitem, the item can be corresponding deleted from the shopping list,i.e., the shopping list can be updated in real time according to thetake-up action or the put-back action of the customer.

A method to judge whether the action on the item is the take-up actionor the put-back action may adopt the following steps.

A motion track of a hand is established on a timeline according to aplurality of collected frames of consecutive hand images of the customerin front of a shelf, and whether the customer applies the take-up actionor the put-back action to the item is judged according to the handmotion track. For example, a camera is deployed at the upper portion ofthe shelf and a shooting angle of the camera is downward, so that ashooting range of the camera covers the shelf. Preferably, a pluralityof cameras are provided, so that multi-angle shooting can be guaranteedand an accuracy of identifying item is improved. The camera can shoot aplurality of frames of images per second, e.g., 30 frames. The handimages of the customer captured by the camera are detected frame byframe. The position of the hand in each frame of hand image is markedand saved. The above-mentioned operations are repeated for every frame.In this way, one motion track of the hand can be obtained on thetimeline. Not only the position of the hand in each frame of image canbe obtained, but also whether the item is taken in the hand can bejudged and a category of the item can be confirmed according to acertain frame of hand image.

Specifically, a virtual action boundary is marked in the hand image, andthe action boundary is spaced a certain distance, such as 10 cm or 20 cmfrom the shelf. in case that it is detected that the motion track of thehand is a movement from a position away from the shelf through theaction boundary to a position close to the shelf, briefly, a movementfrom the outside of the action boundary to the inside of the actionboundary, and the item is taken in the hand, then it is considered thatthe hand action is a put-back action on the item; and in case that it isdetected that the motion track of the hand is a movement from a positionclose to the shelf through the action boundary to a position away fromthe shelf, briefly, a movement from the inside of the action boundary tothe outside of the action boundary, and the item is taken in the hand,then it is considered that the hand action is a take-up action on theitem.

The camera continuously captures video data, shoots a plurality of (suchas dozens of) frames of hand images per second. One frame of hand imageat an interval of one second can be obtained, so as to cut out a handposition, and classify (or called as identify) the item in the hand.This can be implemented by a pre-trained classification model describedbelow.

With reference to FIG. 2, a method for performing identification on theitem at which the take-up action or the put-back action aims can adoptthe following steps of S31 to S35.

In the step S31, target detection on the plurality of frames of handimages containing the item is performed so as to obtain a plurality ofrectangular area images correspondingly, in which the rectangular areaimages are images corresponding to rectangular areas containing theitem, and the plurality of frames of hand images correspond to aplurality of cameras in a one-to-one manner.

Specifically, when target detection is carried out on the hand image, arectangular case (or called as the rectangular area) containing the itemcan be marked out on the hand image. An image corresponding to therectangular case is an image for performing classification on the item.In order to acquire multiple frames of hand images, a plurality ofcameras needs to be arranged. The camera can be arranged right above theitem, and in this case, the camera shoots downwards from right above.The cameras can also be arranged obliquely above the item, and in thiscase, the cameras shoot the item obliquely downwards. Alternatively, apart of the cameras can be arranged right above the item, and the otherpart of the cameras can be arranged obliquely above the item. It shouldbe noted that no matter where the cameras are arranged, a distance fromeach camera to the ground can be either equal or unequal, which is notlimited in the embodiment.

In the step S32, a plurality of primary classification results isacquired correspondingly, according to the plurality of rectangular areaimages and a pre-trained first-level classification model, and afirst-level classification result of the plurality of frames of handimages is acquired according to the plurality of primary classificationresults and a pre-trained first-level linear regression model, in whichthe pre-trained first-level classification model is a model that isconstructed by an image identification technique of convolutional neuralnetwork and trained by all items in the shopping place.

Specifically, data are collected in advance to establish a data set, andthe collecting of the data comprises: 1) shooting all items in theshopping place from each angle and in each attitude to acquire a greatamount of photos; and 2) labeling those photos, i.e., labelingpositions, sizes and categories of the items in the photos. The dataincluded in the data set means the above-mentioned photos and labels onthose photos. The first-level classification model is a modelconstructed by an image identification technique of convolutional neuralnetwork, and is trained by using the data of all items in the shoppingplace. The training can be carried out in a gradient descent mode.

The trained first-level classification model classifies the item in eachrectangular area image to obtain the primary classification result. Theprimary classification result is an n-dimensional vector, where nrepresents the total number of items in a shopping place. Each elementin the vector represents a probability that the to-be-classified itembelongs to each of the n items according to the first-levelclassification model. When the element has a maximum value in thevector, it means that the to-be-classified item is an item correspondingto the element according to the model. When there are five rectangulararea images, the number of the primary classification results is 5n-dimensional vectors.

When the first-level classification model is trained, the primaryclassification results outputted by the first-level classification modelare used as inputs of the first-level linear regression model, andcorrect classifications of the items included in the hand imagescorresponding to the primary classification results are used as outputsof the first-level linear regression model, so as to train thefirst-level linear regression model. The trained first-level linearregression model carries out data fusion on the plurality of primaryclassification results to obtain one first-level classification result.The first-level classification result represents that the first-levellinear regression model predicts to which category of item in theshopping place the item in the image belongs.

In the step S33, the first-level classification result is used as thefirst classification result.

There are various items in the shopping place. Among the various items,there may be some items which are similar in appearance and may easilybe visually confused. Such items are called as similar items, e.g.,Golden Delicious apples and yellow snowflake pears. If singleto-be-classified item is similar items, the first-level classificationmodel is difficult to accurately classify the items. For example, GoldenDelicious apples are confused with yellow snowflake pears and GoldenDelicious apples are classified as yellow snowflake pears. Thus, withreference to FIG. 3, after the step S32, the undermentioned step S35needs to be executed; otherwise, the step S33 is executed, i.e., thefirst-level classification result is directly used as the firstclassification result for checkout.

Specifically, in the step S35, if the first-level classification resultis similar items, then a plurality of secondary classification resultsare correspondingly acquired according to the plurality of rectangulararea images and a pre-trained second-level classification model, asecond-level classification result of the plurality of frames of handimages is acquired according to the plurality of secondaryclassification results and a pre-trained second-level linear regressionmodel, and the second-level classification result is used as a firstclassification result. The second-level classification model is a modelthat is constructed by the image identification technique ofconvolutional neural network and trained by items in a similar itemgroup in the shopping place in advance.

Specifically, the second-level classification model is trained byutilizing data of the similar items in the data set established in thestep S32, and the training can be carried out in a gradient descentmode. The second-level classification model differs from the first-levelclassification model in that different data are used in the trainingprocess, in which the data used by the first-level classification modelare data of all the items in the shopping place, and the data used bythe second-level classification model are the data of the similar itemsin the shopping place.

The trained second-level classification model classifies the item ineach rectangular area image to obtain the secondary classificationresult. The secondary classification result is also a m-dimensionalvector, and each element in the vector represents a probability that thesingle to-be-classified item belongs to each of m similar itemsaccording to the second-level classification model. When there are fiverectangular area images, the number of the secondary classificationresults is 5 m-dimensional vectors, where m is smaller than or equal ton and represents the total number of the similar items in the shoppingplace.

In practice, there are multiple groups of similar items in the shoppingplace. For example, one group of similar items comprise Golden Deliciousapples and yellow snowflake pears, another group of similar itemscomprise loose-packed salt and loose-packed white sugar, and yet anothergroup of similar items comprise dietary alkali and flour. Onesecond-level classification model can be trained for all the groups ofsimilar items. In order to further improve accuracy of itemclassification, one second-level classification model is trained foreach group of similar items. At this point, if the first-levelclassification result is similar items, the second-level classificationmodel corresponding to the first-level classification result is used.

When the second-level classification model is trained, the secondaryclassification results outputted by the second-level classificationmodel are used as inputs of the second-level linear regression model,and correct classifications of the items included in the imagescorresponding to the secondary classification results are used asoutputs of the second-level linear regression model, so as to train thesecond-level linear regression model. The trained second-level linearregression model carries out data fusion on the plurality of secondaryclassification results to obtain one second-level classification result,and the second-level classification result is used as a firstclassification result. The second-level classification result representsthat the second-level linear regression model predicts to which categoryof item in the shopping place the item in the image belongs.

In the step S34, the first classification result is used as theto-be-identified item.

After the first classification result is acquired, an item pricecorresponding to the first classification result is acquired, and thusthe cost that the customer needs to pay for the selected item isdetermined.

In the step S4, the checkout of the shopping list of the customer isperformed, after the customer left the shopping place.

When the customer selected the items and left the shopping place throughthe door of the shopping place from inside to outside, the customer isdetermined to be in a state of leaving the shopping place, and thecheckout is performed according to the shopping list of the customer.For example, the cost corresponding to the shopping list is deductedfrom the payment account inputted upon registration of the customer.

In order to facilitate verification of the customer on the purchaseditems, a identification result can also be sent to the customer in realtime. For example, the identification result of each item can beuploaded to a cloud server. Then the cloud server issues theidentification result to an App installed in a mobile phone of thecustomer. The App adds the identification result into a virtual shoppingcart and generates the shopping list so as to notify the customer of theshopping list immediately after the item is taken up or put back. Whenthe customer came to a doorway of the store and intends to leave theshopping place, the final payment is completed at the doorway of thestore.

With reference to FIG. 4, another embodiment of the present inventionprovides a checkout device, comprising:

a registration module 401, which is used for receiving identityinformation inputted by a customer upon registration and acquiring theidentity information of the customer who intends to enter a shoppingplace;

a real-time tracking module 402, which is connected with theregistration module 401 and used for tracking the customer, whoseidentity information has been acquired by the registration module, inthe shopping place in real time, and acquiring a position of thecustomer;

a shopping list generation module 403, which is connected with thereal-time tracking module 402 and used for judging whether the positionof the customer, which is acquired by the real-time tracking module, isconsistent with a position of an item in the shopping place, and if so,the shopping list generation module associates the customer with atake-up action or a put-back action aiming at the item, and the shoppinglist generation module generates a shopping list of the customer, afteridentifying the take-up action or the put-back action and the item atwhich the take-up action or the put-back action aims; and

a checkout module 404, which is connected with the shopping listgeneration module 403 and used for performing checkout of the shoppinglist generated by the shopping list generation module 403.

Specifically, the shopping list generation module 403 comprises: anassociation unit configured to represent the position of the item with aposition of a forward camera, which is mounted on a shelf for bearingthe item and used for shooting forwards, and in case that the identityinformation of the customer represented by an image containing thecustomer, which is shot by the forward camera, is the same as theidentity information acquired by the registration module, determine thatthe position of the customer is consistent with the position of the itemin the shopping place; an action identification unit configured toacquire a plurality of frames of consecutive hand images of the customerin front of the shelf for bearing the item, and establish a motion trackof the hand for the plurality of frames of consecutive hand images on atimeline, in case that it is detected that the motion track of the handis an inward movement from the outside of a predetermined virtual actionboundary and the item is taken in the hand, the action identificationunit identifies the action as the put-back action; in case that it isdetected that the motion track of the hand is an outward movement fromthe inside of the virtual action boundary and the item is taken in thehand, the action identification unit identifies the action as thetake-up action, wherein the outside of the virtual action boundary is ina direction away from the shelf, and the inside of the virtual actionboundary is in a direction close to the shelf; an item identificationunit configured to identify the item at which the take-up action or theput-back action aims; and a shopping list generation unit configured togenerate the shopping list of the customer, according to the identityinformation of the customer determined by the association unit, thetake-up action or the put-back action identified by the actionidentification unit, and the item at which the take-up action or theput-back action aims and identified by the item identification unit.

Specifically, the item identification unit comprises: a target detectionsubunit configured to perform target detection on the plurality offrames of hand images containing the item which are acquired by theaction identification unit, in order to obtain a plurality ofrectangular area images correspondingly, wherein the rectangular areaimages are images corresponding to rectangular areas containing theitem, and the plurality of frames of hand images correspond to aplurality of cameras in a one-to-one manner; a first classificationsubunit configured to acquire a plurality of primary classificationresults correspondingly, according to the plurality of rectangular areaimages and a pre-trained first-level classification model, and acquire afirst-level classification result of the plurality of frames of handimages according to the plurality of primary classification results anda pre-trained first-level linear regression model, wherein thepre-trained first-level classification model is a model that isconstructed by an image identification technique of convolutional neuralnetwork and trained by all the items in the shopping place; aconfirmation unit configured to use the first-level classificationresult as a first classification result; and a result determination unitconfigured to use the first classification result as a to-be-identifieditem.

It should be noted that the specific description on the registrationmodule 401 can refer to the corresponding content of the step S1 in theabove-mentioned embodiment, the specific description on the verificationassociation module 402 can refer to the corresponding content of thestep S2 in the above-mentioned embodiment, and the specific descriptionon the shopping list generation module 403 can refer to thecorresponding content of the step S3 and the steps S31, S32, S33, S34and S35 in the above-mentioned embodiment, which are not repeatedherein.

Still a further embodiment of the present invention provides a checkoutdevice based on an image identification technology of convolutionalneural network, comprising: a top camera, a forward camera 51, a lowercamera 52, a processor and a memory.

The top camera is used for shooting downwards from the top of a shoppingplace to track a customer, whose identity information has been acquired,in the shopping place in real time. The forward camera is used forshooting towards the front of a shelf to acquire an image of thecustomer positioned in front of the shelf for bearing the item. Thelower camera is used for shooting downwards to acquire a hand image ofthe customer. The memory is configured to record processor-executableinstructions. The processor is configured to identify the pre-registeredcustomer to acquire the identity information containing face data and apayment account of the customer, control the top camera to track thecustomer, whose identity information has been acquired, in real time andacquire a position of the customer, judge whether the position of thecustomer is consistent with a position of the item in the shoppingplace, which is acquired by controlling the forward camera, and if so,associate the customer with a take-up action or a put-back action aimingat the item, and generate a shopping list of the customer afteridentifying the take-up action or the put-back action, and the item atwhich the take-up action or the put-back action aims according to thehand image acquired by the lower camera; and performing checkout of theshopping list.

With reference to FIG. 5, in order to facilitate an accurateidentification and more clear and accurate observation for shooting inthe checkout method, the checkout device and the undermentioned checkoutsystem, an arrangement of cargoes on each bearing platform 53 isdescribed as follows. A shelf (or called as a shelf for the checkoutdevice) for bearing the items comprises: a base 56, an upright 55 andplatforms 53. The base 56 is used for providing a support and arrangedon the ground. The upright 55 is arranged on the base 56. The upright 55may be arranged in a vertical mode. For example, The upright 55 may bearranged at one end of the base 56, so that a combination of the upright55 and the base 56 is L shape, or may also be arranged at the middle ofthe upper surface of the base 56, so that the combination of the upright55 and the base 56 is an inverted T shape, or may be arranged in aninclined mode, which is not limited in the embodiment. A plurality ofbearing platforms 53 are sequentially arranged on the same side of theupright 55 in a vertical direction (when the upright 55 is verticallyarranged on the base 56, the vertical direction is a length direction ofthe upright 55), and an interval is reserved between any two adjacentbearing platforms 53, so as to form a space for accommodatingto-be-placed cargoes, and the cargoes are placed on each of the bearingplatforms 53. One end of the bearing platform 53 away from the upright55 is a free end. In any two adjacent bearing platforms 53, the free endof the upper bearing platform 55 is closer to the upright 55 than thefree end of the lower bearing platform, i.e., widths (i.e., the lengthsof the bearing platforms 53 in a horizontal direction in FIG. 5) of aplurality of bearing platforms 53 are increased gradually from top tobottom. The width of the lowermost bearing platform 53 is the maximum.The width of the uppermost bearing platform 53 is the minimum. In thisway, when shooting the cargoes from top to bottom, the arrangement ofthe cargoes on each of the bearing platforms 53 can be more clearly andaccurately observed.

In practice, the shelf can be named according to the number of thebearing platforms 53. When a plurality of bearing platforms 53 areprovided, the shelf can be called as a multi-layer shelf. Preferably,the number of the bearing platforms 53 is 3 or 4.

The bearing platforms 53 may be a flat plate. The bearing platforms 53are a continuous body. When the cargoes are placed, the cargoes areplaced on the bearing platforms 53, so as to facilitate placement of thecargoes which are heavy and difficult to hang.

In other embodiments, the bearing platform 53 may comprise: a cross barand a plurality of hanging rods. The cross bar is horizontally arrangedon the upright 55. The plurality of hanging rods are vertically arrangedin parallel at intervals on the cross bar. At this point, the bearingplatforms 53 are an intermittent body. When the cargoes are placed, thecargoes are hung below the hanging rods, i.e., the cargoes arepositioned below the bearing platforms 53, so as to facilitate theplacement of cargoes which are lightweight and easy to deform inpackaging.

The shelf is specifically applicable to the shopping place of “Just WalkOut”. The forward camera 51 and the lower camera 52 are arranged at theupper portion of the upright 55. The forward camera 51 shoots towardsthe front of the shelf (e.g., the left side in FIG. 5), i.e., shootingthe customer who is positioned in front of the shelf and selectingcargoes. The lower camera 52 is positioned above the bearing platforms53. The lower camera 52 shoots downwards from the upper portion of thebearing platforms 53, i.e., shooting the cargoes on the bearingplatforms 53. The shooting range of the camera covers the cargoes on theshelf.

Preferably, a plurality of the lower cameras 52 are provided, so that itis ensured that the cargoes selected by the customer may be shot. Theplurality of the lower cameras 52 may be sequentially distributed abovethe bearing platform 53 along the length direction L of the bearingplatforms 53. The height of each of the lower cameras 52 may be equal orunequal. The plurality of the lower cameras 52 may be sequentiallydistributed above the bearing platforms 53 along the width direction Wof the bearing platforms 53. The height of each of the lower cameras 52may be equal or unequal. A part of the plurality of the lower cameras 52may be sequentially distributed above the bearing platforms 53 along thelength direction L of the bearing platforms 53, and the others may besequentially distributed above the bearing platforms 53 along the widthdirection W of the bearing platforms, which is not limited in theembodiment. Preferably, four lower cameras are provided, in which twolower cameras are sequentially distributed in the length direction L ofthe bearing platforms 53, and the others are sequentially distributed inthe width direction W of the bearing platforms 53.

Another embodiment of the present invention provides a checkout systemcomprising: a client terminal for receiving identity informationinputted by a customer upon registration and sending the identityinformation to a checkout device, and receiving a shopping list issuedby the checkout device; and the above-mentioned checkout device.Particular details are omitted herein.

From the above, the embodiments of the present invention bring out thefollowing beneficial effects.

The operation cost is low, since the workload caused by adhesion of RFIDis saved, compared with a RFID solution. The application range is wide,since the present invention is applicable to any item, and is notrestrained by attributes of the item, such as forms, materials and thelike. The user experience is good, since the customer can acquirecorresponding information immediately after taking an item.

It can be known from common technical knowledge that the presentinvention can be implemented by other embodiments without departing fromthe spirit essence or necessary characteristics of the presentinvention. Therefore, the above-mentioned disclosed embodiments, in allaspects, merely are used for illustration rather than limitation. Allchanges made in the scope of the present invention or the scopeequivalent to the present invention shall fall within the presentinvention.

What is claimed is:
 1. A checkout method comprising the steps of:identifying a pre-registered customer to acquire an identity informationof the customer, the identity information containing face data and apayment account of the customer; tracking the customer whose identityinformation has been acquired, in a shopping place in real time, andacquiring a position of the customer; judging whether the position ofthe customer is consistent with a position of an item in the shoppingplace, and if so, associating the customer with a take-up action or aput-back action aiming at the item, and generating a shopping list ofthe customer after identifying the take-up action or the put-backaction, and the item at which the take-up action or the put-back actionaims, wherein identifying the item at which the take-up action or theput-back action aims comprises the steps of: performing target detectionon the plurality of acquired frames of hand images containing the itemto obtain a plurality of rectangular area images correspondingly,wherein the rectangular area images are images corresponding torectangular areas containing the item, and the plurality of frames ofhand images corresponds to a plurality of cameras in a one-to-onemanner; acquiring a plurality of primary classification resultscorrespondingly, according to the plurality of rectangular area imagesand a pre-trained first-level classification model, and acquiring afirst-level classification result of the plurality of frames of handimages, according to the plurality of primary classification results anda pre-trained first-level linear regression model, wherein thepre-trained first-level classification model is a model that isconstructed by an image identification technique of convolutional neuralnetwork and trained by all the items in the shopping place, when thefirst-level classification model is trained, the primary classificationresults outputted by the first-level classification model are used asinputs of the first-level linear regression model, and correctclassifications of the items included in the hand images correspondingto the primary classification results are used as outputs of thefirst-level linear regression model; using the first-levelclassification result as a first classification result; and using thefirst classification result as a to-be-identified item; and performingcheckout of the shopping list.
 2. The checkout method according to claim1, wherein in the step of judging whether the position of the customeris consistent with a position of an item in the shopping place, and ifso, associating the customer with a take-up action or a put-back actionaiming at the item, and generating a shopping list of the customer afteridentifying the take-up action or the put-back action, and the item atwhich the take-up action or the put-back action aims, judging whetherthe position of the customer is consistent with the position of the itemin the shopping place comprises: representing the position of the itemwith a position of a forward camera which is mounted on a shelf forbearing the item and is used for shooting forwards, in case that theidentity information of the customer represented by an image containingthe customer, which is shot by the forward camera, is the same as theidentity information acquired in the step of identifying apre-registered customer to acquire an identity information of thecustomer, the identity information containing face data and a paymentaccount of the customer, judging that the position of the customer isconsistent with the position of the item in the shopping place.
 3. Thecheckout method according to claim 1, wherein in the step of judgingwhether the position of the customer is consistent with a position of anitem in the shopping place, and if so, associating the customer with atake-up action or a put-back action aiming at the item, and generating ashopping list of the customer after identifying the take-up action orthe put-back action, and the item at which the take-up action or theput-back action aims, identifying the take-up action or the put-backaction comprises: acquiring a plurality of frames of consecutive handimages of the customer in front of the shelf for bearing the item, andestablishing a motion track of a hand for the plurality of frames ofconsecutive hand images on a timeline, in case that it is detected thatthe motion track of the hand is an inward movement from the outside of apredetermined virtual action boundary and the item is taken in the hand,identifying the action as the put-back action; in case that it isdetected that the motion track of the hand is an outward movement fromthe inside of the virtual action boundary and the item is taken in thehand, identifying the action as the take-up action, wherein the outsideof the virtual action boundary is in a direction away from the shelf,and the inside of the virtual action boundary is in a direction close tothe shelf.
 4. The checkout method according to claim 1, wherein afterthe step of acquiring a plurality of primary classification resultscorrespondingly, according to the plurality of rectangular area imagesand a pre-trained first-level classification model, and acquiring afirst-level classification result of the plurality of frames of handimages, according to the plurality of primary classification results anda pre-trained first-level linear regression model, wherein thepre-trained first-level classification model is a model that isconstructed by an image identification technique of convolutional neuralnetwork and trained by all the items in the shopping place and beforethe step of using the first classification result as a to-be-identifieditem, the method further comprises the step of: obtaining a plurality ofsecondary classification results correspondingly, according to theplurality of rectangular area images and a pre-trained second-levelclassification model, acquiring a second-level classification result ofthe plurality of frames of hand images, according to the plurality ofsecondary classification results and a pre-trained second-level linearregression model, and using the second-level classification result asthe first classification result, in case that the first-levelclassification result is a similar item, wherein the second-levelclassification model being a model that is constructed by the imageidentification technique of convolutional neural network and trained byitems in a similar item group in the shopping place in advance;otherwise, executing the step of using the first-level classificationresult as a first classification result.
 5. A checkout devicecomprising: a registration module configured to receive identityinformation inputted by a customer upon registration and acquire theidentity information of the customer who intends to enter a shoppingplace; a real-time tracking module configured to be connected with theregistration module and configured to track the customer, whose identityinformation has been acquired by the registration module, in theshopping place in real time, and acquire a position of the customer; ashopping list generation module configured to be connected with thereal-time tracking module and configured to judge whether the positionof the customer acquired by the real-time tracking module is consistentwith a position of an item in the shopping place, and if so, associatethe customer with a take-up action or a put-back action aiming at theitem, and generate a shopping list of the customer, after identifyingthe take-up action or the put-back action and the item at which thetake-up action or the put-back action aims; and a checkout moduleconfigured to be connected with the shopping list generation module andconfigured to perform checkout of the shopping list generated by theshopping list generation module; wherein the shopping list generationmodule comprises an item identification unit configured to identify theitem at which the take-up action or the put-back action aims; the itemidentification unit comprises: a target detection subunit configured toperform target detection on the plurality of frames of hand imagescontaining the item which are acquired by the action identificationunit, in order to obtain a plurality of rectangular area imagescorrespondingly, wherein the rectangular area images are imagescorresponding to rectangular areas containing the item, and theplurality of frames of hand images correspond to a plurality of camerasin a one-to-one manner; a first classification subunit configured toacquire a plurality of primary classification results correspondingly,according to the plurality of rectangular area images and a pre-trainedfirst-level classification model, and acquire a first-levelclassification result of the plurality of frames of hand imagesaccording to the plurality of primary classification results and apre-trained first-level linear regression model, wherein the pre-trainedfirst-level classification model is a model that is constructed by animage identification technique of convolutional neural network andtrained by all the items in the shopping place, when the first-levelclassification model is trained, the primary classification resultsoutputted by the first-level classification model are used as inputs ofthe first-level linear regression model, and correct classifications ofthe items included in the hand images corresponding to the primaryclassification results are used as outputs of the first-level linearregression model; a confirmation unit configured to use the first-levelclassification result as a first classification result; and a resultdetermination unit configured to use the first classification result asa to-be-identified item.
 6. The checkout device according to claim 5,wherein the shopping list generation module comprises: an associationunit configured to represent the position of the item with a position ofa forward camera, which is mounted on a shelf for bearing the item andused for shooting forwards, in case that the identity information of thecustomer represented by an image containing the customer, which is shotby the forward camera, is the same as the identity information acquiredby the registration module, judge that the position of the customer isconsistent with the position of the item in the shopping place; anaction identification unit configured to acquire a plurality of framesof consecutive hand images of the customer in front of the shelf forbearing the item, and establish a motion track of the hand for theplurality of frames of consecutive hand images on a timeline, in casethat it is detected that the motion track of the hand is an inwardmovement from the outside of a predetermined virtual action boundary andthe item is taken in the hand, identify the action as the put-backaction; in case that it is detected that the motion track of the hand isan outward movement from the inside of the virtual action boundary andthe item is taken in the hand, identify the action as the take-upaction, wherein the outside of the virtual action boundary is in adirection away from the shelf, and the inside of the virtual actionboundary is in a direction close to the shelf; and a shopping listgeneration unit configured to generate the shopping list of thecustomer, according to the identity information of the customerdetermined by the association unit, the take-up action or the put-backaction identified by the action identification unit, and the item atwhich the take-up action or the put-back action aims and identified bythe item identification unit.
 7. A checkout device comprising: a topcamera configured to shoot downwards from the top of a shopping place totrack a customer, whose identity information has been acquired, in theshopping place in real time; a forward camera configured to shoottowards the front of a shelf to acquire an image of the customerpositioned in front of the shelf for bearing an item; a lower cameraconfigured to shoot downwards to acquire a hand image of the customer; aprocessor; and a memory that records processor-executable instructions,wherein the processor is configured to: identify the pre-registeredcustomer to acquire the identity information containing face data and apayment account of the customer; control the top camera to track thecustomer, whose identity information has been acquired, in real time andacquire a position of the customer; judge whether the position of thecustomer is consistent with a position of the item in the shoppingplace, which is acquired by controlling the forward camera, and if so,associate the customer with a take-up action or a put-back action aimingat the item, and generate a shopping list of the customer afteridentifying the take-up action or the put-back action and the item atwhich the take-up action or the put-back action aims according to thehand image acquired by the lower camera, wherein identifying the item atwhich the take-up action or the put-back action aims comprises the stepsof: performing target detection on the plurality of acquired frames ofhand images containing the item to obtain a plurality of rectangulararea images correspondingly, wherein the rectangular area images areimages corresponding to rectangular areas containing the item, and theplurality of frames of hand images corresponds to a plurality of camerasin a one-to-one manner; acquiring a plurality of primary classificationresults correspondingly, according to the plurality of rectangular areaimages and a pre-trained first-level classification model, and acquiringa first-level classification result of the plurality of frames of handimages, according to the plurality of primary classification results anda pre-trained first-level linear regression model, wherein thepre-trained first-level classification model is a model that isconstructed by an image identification technique of convolutional neuralnetwork and trained by all the items in the shopping place, when thefirst-level classification model is trained, the primary classificationresults outputted by the first-level classification model are used asinputs of the first-level linear regression model, and correctclassifications of the items included in the hand images correspondingto the primary classification results are used as outputs of thefirst-level linear regression model; using the first-levelclassification result as a first classification result; and using thefirst classification result as a to-be-identified item; and performcheckout of the shopping list.
 8. A checkout system comprising: a clientterminal configured to receive identity information inputted by acustomer upon registration and send the identity information to acheckout device, and configured to receive a shopping list issued by thecheckout device; and the checkout device comprising: a registrationmodule configured to receive identity information inputted by a customerupon registration and acquire the identity information of the customerwho intends to enter a shopping place; a real-time tracking moduleconfigured to be connected with the registration module and configuredto track the customer, whose identity information has been acquired bythe registration module, in the shopping place in real time, and acquirea position of the customer; a shopping list generation module configuredto be connected with the real-time tracking module and configured tojudge whether the position of the customer acquired by the real-timetracking module is consistent with a position of an item in the shoppingplace, and if so, associate the customer with a take-up action or aput-back action aiming at the item, and generate a shopping list of thecustomer, after identifying the take-up action or the put-back actionand the item at which the take-up action or the put-back action aims;and a checkout module configured to be connected with the shopping listgeneration module and configured to perform checkout of the shoppinglist generated by the shopping list generation module; wherein theshopping list generation module comprises an item identification unitconfigured to identify the item at which the take-up action or theput-back action aims; the item identification unit comprises: a targetdetection subunit configured to perform target detection on theplurality of frames of hand images containing the item which areacquired by the action identification unit, in order to obtain aplurality of rectangular area images correspondingly, wherein therectangular area images are images corresponding to rectangular areascontaining the item, and the plurality of frames of hand imagescorrespond to a plurality of cameras in a one-to-one manner; a firstclassification subunit configured to acquire a plurality of primaryclassification results correspondingly, according to the plurality ofrectangular area images and a pre-trained first-level classificationmodel, and acquire a first-level classification result of the pluralityof frames of hand images according to the plurality of primaryclassification results and a pre-trained first-level linear regressionmodel, wherein the pre-trained first-level classification model is amodel that is constructed by an image identification technique ofconvolutional neural network and trained by all the items in theshopping place, when the first-level classification model is trained,the primary classification results outputted by the first-levelclassification model are used as inputs of the first-level linearregression model, and correct classifications of the items included inthe hand images corresponding to the primary classification results areused as outputs of the first-level linear regression model; aconfirmation unit configured to use the first-level classificationresult as a first classification result; and a result determination unitconfigured to use the first classification result as a to-be-identifieditem.